go-cve-dictionary

This is tool to build a local copy of the NVD (National Vulnerabilities Database) [1] and the Japanese JVN [2], which contain security vulnerabilities according to their CVE identifiers [3] including exhaustive information and a risk score. The local copy is generated in sqlite format, and the tool has a server mode for easy querying.

[1] https://en.wikipedia.org/wiki/National_Vulnerability_Database
[2] https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures
[3] http://jvndb.jvn.jp/apis/termsofuse.html

Installation

Install requirements

go-cve-dictionary requires the following packages.

• SQLite3, MySQL, PostgreSQL or Redis
• git
• gcc
• go v1.7.1 or later
  • https://golang.org/doc/install

Here's an example for Amazon EC2 server.

$ ssh ec2-user@52.100.100.100  -i ~/.ssh/private.pem
$ sudo yum -y install sqlite git gcc
$ wget https://storage.googleapis.com/golang/go1.7.1.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xzf go1.7.1.linux-amd64.tar.gz
$ mkdir $HOME/go

Put these lines into /etc/profile.d/goenv.sh

export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin

Set the OS environment variable to current shell

$ source /etc/profile.d/goenv.sh

Deploy go-cve-dictionary

To install:

$ mkdir -p $GOPATH/src/github.com/vulsio
$ cd $GOPATH/src/github.com/vulsio
$ git clone https://github.com/vulsio/go-cve-dictionary.git
$ cd go-cve-dictionary
$ make install

Create a log output directory. You can use another directory on the command line option (--log-dir).

$ sudo mkdir /var/log/go-cve-dictionary
$ sudo chown ec2-user /var/log/go-cve-dictionary
$ sudo chmod 700 /var/log/go-cve-dictionary

Fetch vulnerability data from NVD.

$ go-cve-dictionary fetch nvd
... snip ...
$ ls -alh cve.sqlite3
-rw-r--r-- 1 ec2-user ec2-user 7.0M Mar 24 13:20 cve.sqlite3

Now we have vulnerability data. Start go-cve-dictionary as server mode.

$ go-cve-dictionary server
[Mar 24 15:21:55]  INFO Opening DB. datafile: /home/ec2-user/cve.sqlite3
[Mar 24 15:21:55]  INFO Migrating DB
[Mar 24 15:21:56]  INFO Starting HTTP Sever...
[Mar 24 15:21:56]  INFO Listening on 127.0.0.1:1323

Update go-cve-dictionary

If the DB schema was changed, please specify new SQLite3, MySQL, PostgreSQL or Redis DB file.

$ cd $GOPATH/src/github.com/vulsio/go-cve-dictionary
$ git pull
$ rm -r vendor
$ make install

Binary files are created under $GOPATH/bin

---

Sample data sources

Hello HeartBleed

$ curl http://127.0.0.1:1323/cves/CVE-2014-0160 | jq "."
{
  "CveID": "CVE-2014-0160",
  "Nvd": {
    "Summary": "The (1) TLS and (2) DTLS implementations in OpenSSL 1.0.1 before 1.0.1g do not properly handle Heartbeat Extension packets, which allows remote attackers to obtain sensitive information from process memory via crafted packets that trigger a buffer over-read, as demonstrated by reading private keys, related to d1_both.c and t1_lib.c, aka the Heartbleed bug.",
    "Score": 5,
    "AccessVector": "NETWORK",
    "AccessComplexity": "LOW",
    "Authentication": "NONE",
    "ConfidentialityImpact": "PARTIAL",
    "IntegrityImpact": "NONE",
    "AvailabilityImpact": "NONE",
    "Cpes": null,
    "References": [
      {
        "Source": "CERT",
        "Link": "http://www.us-cert.gov/ncas/alerts/TA14-098A"
      },
      ...snip...
    ],
    "PublishedDate": "2014-04-07T18:55:03.893-04:00",
    "LastModifiedDate": "2015-10-22T10:19:38.453-04:00"
  },
  "Jvn": {
    "Title": "OpenSSL の heartbeat 拡張に情報漏えいの脆弱性",
    "Summary": "OpenSSL の heartbeat 拡張の実装には、情報漏えいの脆弱性が存在します。TLS や DTLS 通信において OpenSSL のコードを実行しているプロセスのメモリ内容が通信相手に漏えいする可能性があります。",
    "JvnLink": "http://jvndb.jvn.jp/ja/contents/2014/JVNDB-2014-001920.html",
    "JvnID": "JVNDB-2014-001920",
    "Score": 5,
    "Severity": "Medium",
    "Vector": "(AV:N/AC:L/Au:N/C:P/I:N/A:N)",
    "References": [
      {
        "Source": "AT-POLICE",
        "Link": "http://www.npa.go.jp/cyberpolice/detect/pdf/20140410.pdf"
      },
      ...snip...
    ],
    "Cpes": null,
    "PublishedDate": "2014-04-08T16:13:59+09:00",
    "LastModifiedDate": "2014-04-08T16:13:59+09:00"
  }
}

Hello Ruby on Rails 4.0.2

$ curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"name": "cpe:/a:rubyonrails:ruby_on_rails:4.0.2:-"}' http://localhost:1323/cpes | jq "."
[
  {
    "CveID": "CVE-2016-0751",
    "Nvd": {
      "CveDetailID": 345,
      "Summary": "actionpack/lib/action_dispatch/http/mime_type.rb in Action Pack in Ruby on Rails before 3.2.22.1, 4.0.x and 4.1.x before 4.1.14.1, 4.2.x before 4.2.5.1, and 5.x before 5.0.0.beta1.1 does not properly restrict use of the MIME type cache, which allows remote attackers to cause a denial of service (memory consumption) via a crafted HTTP Accept header.",
      "Score": 5,
      "AccessVector": "NETWORK",
      "AccessComplexity": "LOW",
      "Authentication": "NONE",
      "ConfidentialityImpact": "NONE",
      "IntegrityImpact": "NONE",
      "AvailabilityImpact": "PARTIAL",
      "Cpes": null,
      "References": [
        {
          "Source": "MLIST",
          "Link": "https://groups.google.com/forum/message/raw?msg=ruby-security-ann/9oLY_FCzvoc/5CDXbvpYEgAJ"
        },
        {
          "Source": "MLIST",
          "Link": "http://www.openwall.com/lists/oss-security/2016/01/25/9"
        }
      ],
      "PublishedDate": "2016-02-15T21:59:05.877-05:00",
      "LastModifiedDate": "2016-03-18T21:02:43.817-04:00"
    },
    "Jvn": {
      "Title": "",
      "Summary": "",
      "JvnLink": "",
      "JvnID": "",
      "Score": 0,
      "Severity": "",
      "Vector": "",
      "References": null,
      "Cpes": null,
      "PublishedDate": "0001-01-01T00:00:00Z",
      "LastModifiedDate": "0001-01-01T00:00:00Z"
    }
  },
  ... snip ...
]

---

Usage

$ go-cve-dictionary --help
GO CVE Dictionary

Usage:
  go-cve-dictionary [command]

Available Commands:
  completion  generate the autocompletion script for the specified shell
  fetch       Fetch Vulnerability dictionary
  help        Help about any command
  search      Search for Vulnerability in the dictionary
  server      Start CVE dictionary HTTP Server
  version     Show version

Flags:
      --config string       config file (default is $HOME/.go-cve-dictionary.yaml)
      --dbpath string       /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
      --dbtype string       Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
      --debug               debug mode (default: false)
      --debug-sql           SQL debug mode
  -h, --help                help for go-cve-dictionary
      --http-proxy string   http://proxy-url:port (default: empty)
      --log-dir string      /path/to/log (default "/var/log/go-cve-dictionary")
      --log-json            output log as JSON
      --log-to-file         output log to file

Use "go-cve-dictionary [command] --help" for more information about a command.

---

Usage: Fetch Command

$ go-cve-dictionary fetch --help
Fetch Vulnerability dictionary

Usage:
  go-cve-dictionary fetch [command]

Available Commands:
  fortinet    Fetch Vulnerability dictionary from Fortinet Advisories
  jvn         Fetch Vulnerability dictionary from JVN
  nvd         Fetch Vulnerability dictionary from NVD

Flags:
      --batch-size int   The number of batch size to insert. (default 5)
  -h, --help             help for fetch

Global Flags:
      --config string       config file (default is $HOME/.go-cve-dictionary.yaml)
      --dbpath string       /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
      --dbtype string       Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
      --debug               debug mode (default: false)
      --debug-sql           SQL debug mode
      --http-proxy string   http://proxy-url:port (default: empty)
      --log-dir string      /path/to/log (default "/var/log/go-cve-dictionary")
      --log-json            output log as JSON
      --log-to-file         output log to file

Use "go-cve-dictionary fetch [command] --help" for more information about a command.

Fetch NVD data

• to fetch all years

$ go-cve-dictionary fetch nvd

• to fetch specific years

$ go-cve-dictionary fetch nvd 2021

Fetch JVN data

• to fetch all years

$ go-cve-dictionary fetch jvn

• to fetch specific years

$ go-cve-dictionary fetch jvn 2021

Fetch Fortinet data

$ go-cve-dictionary fetch fortinet

---

Usage: Run HTTP Server

$ go-cve-dictionary server --help
Start CVE dictionary HTTP Server

Usage:
  go-cve-dictionary server [flags]

Flags:
      --bind string   HTTP server bind to IP address (default "127.0.0.1")
  -h, --help          help for server
      --port string   HTTP server port number (default "1323")

Global Flags:
      --config string       config file (default is $HOME/.go-cve-dictionary.yaml)
      --dbpath string       /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
      --dbtype string       Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
      --debug               debug mode (default: false)
      --debug-sql           SQL debug mode
      --http-proxy string   http://proxy-url:port (default: empty)
      --log-dir string      /path/to/log (default "/var/log/go-cve-dictionary")
      --log-json            output log as JSON
      --log-to-file         output log to file

---

Usage: Search Command

$ go-cve-dictionary search --help
Search for Vulnerability in the dictionary

Usage:
  go-cve-dictionary search [command]

Available Commands:
  cpe         Search for Vulnerability in the dictionary by CPE
  cve         Search for Vulnerability in the dictionary by CVEID

Flags:
  -h, --help   help for search

Global Flags:
      --config string       config file (default is $HOME/.go-cve-dictionary.yaml)
      --dbpath string       /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
      --dbtype string       Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
      --debug               debug mode (default: false)
      --debug-sql           SQL debug mode
      --http-proxy string   http://proxy-url:port (default: empty)
      --log-dir string      /path/to/log (default "/var/log/go-cve-dictionary")
      --log-json            output log as JSON
      --log-to-file         output log to file

Use "go-cve-dictionary search [command] --help" for more information about a command.

Search All CVE IDs

$ go-cve-dictionary search cve
[
  "CVE-2023-38624",
  "CVE-2024-20750",
  "CVE-2024-21101",
  "CVE-2023-27427",
  "CVE-2023-30445",
...

Search by CVE ID(s)

$ go-cve-dictionary search cve CVE-2024-3400
{
  "CveID": "CVE-2024-3400",
  "Nvds": [
    {
      "CveID": "CVE-2024-3400",
      "Descriptions": [
        {
          "Lang": "en",
          "Value": "A command injection as a result of arbitrary file creation vulnerability in the GlobalProtect feature of Palo Alto Networks PAN-OS software for specific PAN-OS versions and distinct feature configurations may enable an unauthenticated attacker to execute arbitrary code with root privileges on the firewall.\n\nCloud NGFW, Panorama appliances, and Prisma Access are not impacted by this vulnerability."
        },
...

$ go-cve-dictionary search cve CVE-2023-48783 CVE-2024-3400
{
  "CVE-2023-48783": {
    "CveID": "CVE-2023-48783",
    "Nvds": [
      {
        "CveID": "CVE-2023-48783",
        ...
      }
    ],
    "Jvns": [],
    "Fortinets": [
      {
        "AdvisoryID": "FG-IR-23-408",
        "CveID": "CVE-2023-48783",
        ...
      }
    ]
  },
  "CVE-2024-3400": {
    "CveID": "CVE-2024-3400",
    "Nvds": [
      {
        "CveID": "CVE-2024-3400",
...

Search by CPE

$ go-cve-dictionary search cpe "cpe:/a:fortinet:fortiportal"
[
  {
    "CveID": "CVE-2017-7337",
    "Nvds": [],
    "Jvns": [],
    "Fortinets": [
      {
        "AdvisoryID": "FG-IR-17-114",
        "CveID": "CVE-2017-7337",
        "Title": "FortiPortal Multiple Vulnerabilities",
...

Search CVE IDs by CPE

$ go-cve-dictionary search cpe --cveid-only "cpe:/a:fortinet:fortiportal"
{
  "Fortinet": [
    "CVE-2023-46712",
    "CVE-2023-48791",
    "CVE-2017-7339",
    "CVE-2017-7343",
    "CVE-2022-27490",
    "CVE-2017-7342",
    "CVE-2017-7731",
    "CVE-2023-41842",
    "CVE-2023-48783",
    "CVE-2024-21761",
    "CVE-2017-7337",
    "CVE-2017-7338",
    "CVE-2017-7340"
  ],
  "JVN": [],
  "NVD": [
    "CVE-2023-46712",
    "CVE-2023-48791",
    "CVE-2023-41842",
    "CVE-2023-48783",
    "CVE-2024-21761"
  ]
}

---

Usage: Use MySQL as a DB storage back-end

• fetch nvd

  $ go-cve-dictionary fetch nvd \
        --dbtype mysql \
        --dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"

• fetch jvn

  $ go-cve-dictionary fetch jvn \
        --dbtype mysql \
        --dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"

• fetch fortinet

  $ go-cve-dictionary fetch fortinet \
        --dbtype mysql \
        --dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"

• server

  $ go-cve-dictionary server \
        --dbtype mysql \
        --dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"

Usage: Use Postgres as a DB storage back-end

• fetch nvd

  $ go-cve-dictionary fetch nvd \
        --dbtype postgres \
        --dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"

• fetch jvn

  $ go-cve-dictionary fetch jvn \
        --dbtype postgres \
        --dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"

• fetch fortinet

  $ go-cve-dictionary fetch fortinet \
        --dbtype postgres \
        --dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"

• server

  $ go-cve-dictionary server \
        --dbtype postgres \
        --dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"

Usage: Use Redis as a DB storage back-end

• fetch nvd

  $ go-cve-dictionary fetch nvd \
        --dbtype redis \
        --dbpath "redis://localhost/0"

• fetch jvn

  $ go-cve-dictionary fetch jvn \
        --dbtype redis \
        --dbpath "redis://localhost/0"

• fetch fortinet

  $ go-cve-dictionary fetch fortinet \
        --dbtype redis \
        --dbpath "redis://localhost/0"

• server

  $ go-cve-dictionary server \
        --dbtype redis \
        --dbpath "redis://localhost/0"

---

Misc

• HTTP Proxy Support
  If your system at behind HTTP proxy, you have to specify -http-proxy option.

• How to daemonize go-cve-dictionary
  Use Systemd, Upstart or supervisord, daemontools...

• How to cross compile

  $ cd /path/to/your/local-git-repository/go-cve-dictionary
  $ GOOS=linux GOARCH=amd64 go build -o cvedict.amd64

• Logging
  go-cve-dictionary writes a log under -log-path specified directory (default is /var/log/go-cve-dictionary/).

• Debug
  Run with --debug, --debug-sql option.

• Completion Support

  • bash

  $ go-cve-dictionary completion bash > /usr/share/bash-completion/completions/go-cve-dictionary

  • zsh

  $ go-cve-dictionary completion zsh > ~/.zsh-completions/go-cve-dictionary

  • fish

  $ go-cve-dictionary completion fish > ~/.config/fish/completions/go-cve-dictionary.fish

---

Data Source

• NVD
• JVN(Japanese)
• Fortinet(https://www.fortiguard.com/psirt)

---

Authors

kotakanbe (@kotakanbe) created go-cve-dictionary and these fine people have contributed.

---

How to Contribute

1. fork a repository: github.com/vulsio/go-cve-dictionary to github.com/you/repository
2. get original code: github.com/vulsio/go-cve-dictionary
3. work on original code
4. add remote to your repository: git remote add myfork https://github.com/you/repo.git
5. push your changes: git push myfork
6. create a new Pull Request

• see GitHub and Go: forking, pull requests, and go-getting

---

Licence

Please see LICENSE.

---

Additional License

• NVD

How can my organization use the NVD data within our own products and services? All NVD data is freely available from our XML Data Feeds. There are no fees, licensing restrictions, or even a requirement to register. All NIST publications are available in the public domain according to Title 17 of the United States Code. Acknowledgment of the NVD when using our information is appreciated. In addition, please email nvd@nist.gov to let us know how the information is being used.

• JVN